Image processing device and image processing method

ABSTRACT

A deblocking filter  24  performs filtering of decoded image data obtained by decoding image data encoded for each block, so as to remove block distortion. If at least one of block sizes on adjacent sides of two adjacent blocks is extended larger than a predetermined block size, a filter setting unit  41  sets the tap length to an extended length to increase the strength of distortion removal, or sets a filtering object pixel range to an extended range. When a macroblock having an extended size is used, the degree of smoothing is increased, and pixels including those distant from the block boundary are subjected to filtering. Consequently, even when various block sizes are employed or when blocks of extended sizes are used, images of high image quality can be achieved.

TECHNICAL FIELD

The present technology relates to an image processing device and animage processing method, and particularly makes it possible to obtain adecoded image of excellent image quality.

BACKGROUND ART

In recent years, devices that treat image information as digital andtransmit or store the information with high efficiency, or for exampledevices complying with systems of MPEG and the like in which compressionis performed by an orthogonal transform such as a discrete cosinetransform and motion compensation, have been spreading in broadcastingstations and ordinary households.

MPEG2 (ISO/IEC 13818-2), in particular, is defined as a general-purposeimage coding system, and is now used widely in a wide range ofapplications for professional use and consumer use. The use of thisMPEG2 compression system can achieve excellent image quality byassigning an amount of code (bit rate) of four to eight Mbps in a caseof an interlaced image of a standard resolution with 720×480 pixels, forexample. Excellent image quality can also be achieved by assigning anamount of code (bit rate) of 18 to 22 Mbps in a case of an interlacedimage of a high resolution with 1920×1088 pixels.

MPEG2 was intended for high-image-quality coding to be adapted mainlyfor broadcasting, and did not make provisions for a coding system with alower amount of code (bit rate), that is, a higher compression ratiothan MPEG1. With the spread of portable terminals, a need for such acoding system is expected to increase in the future. Accordingly, anMPEG4 coding system was standardized. A standard for an image codingsystem was approved as an international standard of ISO/IEC 14496-2 inDecember 1998.

Further, standardization of a standard referred to as H.26L (ITU-T Q6/16VCEG) has recently been under way with an original objective of imagecoding for videoconferences. It is known that H.26L requires a largeramount of operation for coding and decoding but achieves a higher codingefficiency compared to conventional coding systems such as MPEG2, MPEG4,and the like. In addition, as part of activities of MPEG4,standardization for achieving higher coding efficiency on the basis ofthis H.26L is now under way as Joint Model of Enhanced-Compression VideoCoding. As for a standardization schedule, an international standard wasestablished under a name of H.264 and MPEG-4 Part 10 (Advanced VideoCoding, hereinafter written as “H.264/AVC”) in March 2003.

Further, as an extension thereof, standardization of FRExt (FidelityRange Extension) including coding tools necessary for business use suchas RGB, 4:2:2 and 4:4:4 as well as an 8×8 DCT and a quantization matrixdefined in MPEG2 was completed in February 2005. Thereby, the H.264/AVCsystem is used as a coding system capable of excellently representingeven film noise included in movies, and is used in a wide range ofapplications such as Blu-Ray (registered trademark).

In such a coding and decoding process, image data is coded in blockunits. In addition, in decoding coded data, as shown in Patent Document1, for example, block distortion is suppressed by performing filteringon the basis of a block boundary strength and a quantization parameter.

Further, there have recently been increasing needs for coding with astill higher compression ratio, such as a desire to compress images ofabout 4000×2000 pixels or a desire to distribute high-definition imagesin an environment with a limited transmission capacity such as theInternet. Thus, as in Non-Patent Document 1, setting a macroblock sizeto a size larger than that of MPEG2 or H.264/AVC, for example a size of32 pixels×32 pixels, is proposed. Specifically, in Non-Patent Document1, a hierarchical structure is adopted for macroblocks, wherebycompatibility with macroblocks in H.264/AVC is maintained for 16×16pixel blocks and smaller blocks, and larger blocks are defined as asuperset thereof.

PRIOR ART DOCUMENTS Patent Document

-   Patent Document 1: Japanese Patent Laid-Open No.

Non-Patent Document

-   Non-Patent Document 1: “Video Coding Using Extended Block” (Study    Group 16, Contribution 123, ITU, January 2009)

SUMMARY OF INVENTION Technical Problem

Incidentally, when block distortion is removed by a conventionaldeblocking filter, the block distortion may be increased at a low bitrate in particular, so that there is a fear of the block distortion notbeing removed sufficiently and image quality being degraded.

It is accordingly an object of the present technology to provide animage processing device and an image processing method that can providean image of excellent image quality even when various block sizes areused or when a block of an extended size is used.

Technical Solutions

According to a first aspect of the present technology, there is providedan image processing device including: a decoding section for decodingimage data coded in each block; a filter for applying filtering forremoving block distortion to the decoded image data decoded by thedecoding section; and a filter setting section for setting, according toblock sizes of adjacent blocks adjacent at a block boundary, a taplength of the filtering for the block boundary or a filtering objectpixel range as an object of the filtering.

In the present technology, a filter for applying filtering for removingblock distortion to decoded image data obtained by decoding image datacoded in each block and a filter setting section for setting the filterare provided. When at least one of adjacent blocks adjacent at a blockboundary is extended larger than a predetermined block size, forexample, the filter setting section sets the tap length of the filteringfor the block boundary longer as the block size is increased or sets afiltering object pixel range as an object of the filtering wider as theblock size is increased. In addition, the tap length of the filtering orthe filtering object pixel range is set according to block sizes onadjacent sides of the adjacent blocks. In addition, case classificationcorresponding to the block sizes of the adjacent blocks is performed,and the tap length of the filtering and the filtering object pixel rangeare set according to a case where the adjacent blocks both have apredetermined block size or smaller and a case where at least one of theadjacent blocks is extended larger than the predetermined block size.The case classification is performed into for example a case where theadjacent blocks are 16×16 pixels or smaller, a case where at least oneof the two blocks is larger than 16×16 pixels and both are 32×32 pixelsor smaller, and a case where at least one of the two blocks is largerthan 32×32 pixels. The block sizes are prediction block sizes asprocessing units when intra prediction or inter prediction is performed.Further, the filter setting section sets the tap length or the filteringobject pixel range according to whether the decoded image data is imagedata for generating a predictive image or image data for image display.

According to a second aspect of the present technology, there isprovided an image processing method including: a decoding step ofdecoding image data coded in each block; a filter step of applyingfiltering for removing block distortion to the decoded image datadecoded in the decoding step; and a filter step of setting, according toblock sizes of adjacent blocks adjacent at a block boundary, a taplength of the filtering for the block boundary or a filtering objectpixel range as an object of the filtering.

According to a third aspect of the present technology, there is providedan image processing device including: a filter for applying filteringfor removing block distortion to decoded image data obtained by locallydecoding image data resulting from an orthogonal transform andquantization; a filter setting section for setting, according to blocksizes of adjacent blocks adjacent at a block boundary, a tap length ofthe filtering for the block boundary or a filtering object pixel rangeas an object of the filtering; and a coding section for performingcoding in each image data block using the decoded image data resultingfrom the filtering by the filter.

According to a fourth aspect of the present technology, there isprovided an image processing method including: a filter step of applyingfiltering for removing block distortion to decoded image data obtainedby locally decoding image data resulting from an orthogonal transformand quantization; a filter setting step of setting, according to blocksizes of adjacent blocks adjacent at a block boundary, a tap length ofthe filtering for the block boundary or a filtering object pixel rangeas an object of the filtering; and a coding step of performing coding ineach image data block using the decoded image data resulting from thefiltering in the filter step.

Advantageous Effect

According to the present technology, an image of excellent image qualitywith a reduced block distortion can be obtained.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram showing a configuration of an image coding device.

FIG. 2 is a diagram showing pixel data used in filtering of a deblockingfilter.

FIG. 3 is a diagram showing relation between a quantization parameter QPand a threshold value α.

FIG. 4 is a diagram showing a configuration of the deblocking filter anda filter setting section.

FIG. 5 is a diagram showing prediction block sizes used in image codingprocesses.

FIG. 6 is a flowchart of an image coding process operation.

FIG. 7 is a flowchart of a predicting process.

FIG. 8 is a flowchart of an intra prediction process.

FIG. 9 is a flowchart of an inter prediction process.

FIG. 10 is a flowchart of a filter setting process.

FIG. 11 shows a configuration of an image decoding device.

FIG. 12 is a flowchart of an image decoding process operation.

FIG. 13 is a diagram illustrating a schematic configuration of atelevision device.

FIG. 14 is a diagram illustrating a schematic configuration of aportable telephone.

FIG. 15 is a diagram illustrating a schematic configuration of arecording and reproducing device.

FIG. 16 is a diagram illustrating a schematic configuration of animaging device.

MODES FOR CARRYING OUT THE INVENTION

A mode for carrying out the present technology will hereinafter bedescribed. An image processing device according to the presenttechnology is applicable to an image coding device for coding image datain a prediction block size, an image decoding device for decoding imagedata coded in a prediction block size, and the like. Thus, a case wherethe image processing device according to the present technology isapplied to an image coding device and a case where the image processingdevice according to the present technology is applied to an imagedecoding device will be described in the following order.

1. Configuration of Image Coding Device

2. Filtering of Deblocking Filter

3. Configuration of Deblocking Filter in Image Coding Device

4. Operation of Image Coding Device

5. Configuration of Image Decoding Device

6. Operation of Image Decoding Device

7. Examples of Application

<1. Configuration of Image Coding Device>

FIG. 1 shows a configuration of an image coding device. The image codingdevice 10 includes an analog/digital converting section (A/D convertingsection) 11, a picture rearrangement buffer 12, a subtracting section13, an orthogonal transform section 14, a quantizing section 15, alossless coding section 16, a storage buffer 17, and a rate controllingsection 18. The image coding device 10 further includes a dequantizingsection 21, an inverse orthogonal transform section 22, an addingsection 23, a deblocking filter 24, a frame memory 25, a selector 26, anintra prediction section 31, a motion prediction and compensationsection 32, and a predictive image and optimum mode selecting section33.

The A/D converting section 11 converts an analog image signal intodigital image data, and outputs the digital image data to the picturerearrangement buffer 12.

The picture rearrangement buffer 12 rearranges frames of the image dataoutput from the A/D converting section 11. The picture rearrangementbuffer 12 rearranges the frames according to a GOP (Group of Pictures)structure involved in a coding process, and outputs the image data afterthe rearrangement to the subtracting section 13, the intra predictionsection 31, and the motion prediction and compensation section 32.

The subtracting section 13 is supplied with the image data output fromthe picture rearrangement buffer 12 and predictive image data selectedby the predictive image and optimum mode selecting section 33 to bedescribed later. The subtracting section 13 calculates prediction errordata indicating differences between the image data output from thepicture rearrangement buffer 12 and the predictive image data suppliedfrom the predictive image and optimum mode selecting section 33, andoutputs the prediction error data to the orthogonal transform section14.

The orthogonal transform section 14 subjects the prediction error dataoutput from the subtracting section 13 to an orthogonal transformprocess such as a Discrete Cosine Transform (DCT), a Karhunen-Loevetransform, or the like. The orthogonal transform section 14 outputstransform coefficient data obtained by performing the orthogonaltransform process to the quantizing section 15.

The quantizing section 15 is supplied with the transform coefficientdata output from the orthogonal transform section 14 and a ratecontrolling signal from the rate controlling section 18 to be describedlater. The quantizing section 15 quantizes the transform coefficientdata, and outputs the quantized data to the lossless coding section 16and the dequantizing section 21. In addition, the quantizing section 15changes a quantization parameter (quantization scale) on the basis ofthe rate controlling signal from the rate controlling section 18 tochange the bit rate of the quantized data.

The lossless coding section 16 is supplied with the quantized dataoutput from the quantizing section 15 and prediction mode informationfrom the intra prediction section 31, the motion prediction andcompensation section 32, and the predictive image and optimum modeselecting section 33 to be described later. Incidentally, the predictionmode information includes a macroblock type enabling the identificationof a prediction block size, a prediction mode, motion vectorinformation, reference picture information, and the like according tointra prediction or inter prediction. The lossless coding section 16subjects the quantized data to a lossless coding process byvariable-length coding or arithmetic coding, for example, therebygenerates a coded stream, and outputs the coded stream to the storagebuffer 17. In addition, the lossless coding section 16 losslessly codesthe prediction mode information, and adds the prediction modeinformation to for example header information of the coded stream.

The storage buffer 17 stores the coded stream from the lossless codingsection 16. In addition, the storage buffer 17 outputs the stored codedstream at a transmission speed corresponding to a transmission line.

The rate controlling section 18 monitors the free space of the storagebuffer 17, generates the rate controlling signal according to the freespace, and outputs the rate controlling signal to the quantizing section15. The rate controlling section 18 for example obtains informationindicating the free space from the storage buffer 17. When the freespace is reduced, the rate controlling section 18 makes the bit rate ofthe quantized data decreased by the rate controlling signal. When thestorage buffer 17 has a sufficiently large free space, the ratecontrolling section 18 makes the bit rate of the quantized data raisedby the rate controlling signal.

The dequantizing section 21 subjects the quantized data supplied fromthe quantizing section 15 to a dequantizing process. The dequantizingsection 21 outputs transform coefficient data obtained by performing thedequantizing process to the inverse orthogonal transform section 22.

The inverse orthogonal transform section 22 outputs data obtained bysubjecting the transform coefficient data supplied from the dequantizingsection 21 to an inverse orthogonal transform process to the addingsection 23.

The adding section 23 generates decoded image data by adding togetherthe data supplied from the inverse orthogonal transform section 22 andthe predictive image data supplied from the predictive image and optimummode selecting section 33, and outputs the decoded image data to thedeblocking filter 24 and the frame memory 25.

The deblocking filter 24 performs filtering to reduce block distortionoccurring at a time of image coding. The deblocking filter 24 performsthe filtering to remove the block distortion from the decoded image datasupplied from the adding section 23, and outputs the decoded image dataafter the filtering to the frame memory 25. In addition, the deblockingfilter 24 sets a tap length and a filtering object pixel range on thebasis of parameter values supplied from a filter setting section 41 tobe described later.

The frame memory 25 retains the decoded image data supplied from theadding section 23 and the decoded image data after the filtering whichdecoded image data is supplied from the deblocking filter 24.

The selector 26 supplies the decoded image data before the filteringwhich decoded image data is read from the frame memory 25 to the intraprediction section 31 to perform intra prediction. In addition, theselector 26 supplies the decoded image data after the filtering whichdecoded image data is read from the frame memory 25 to the motionprediction and compensation section 32 to perform inter prediction.

The intra prediction section 31 performs an intra prediction process inall intra prediction modes as candidates using the image data of acoding object image output from the picture rearrangement buffer 12 andthe decoded image data before the filtering which decoded image data isread from the frame memory 25. Further, the intra prediction section 31calculates a cost function value for each intra prediction mode, andselects an intra prediction mode in which the calculated cost functionvalue is a minimum, that is, an intra prediction mode in which bestcoding efficiency is obtained as an optimum intra prediction mode. Theintra prediction section 31 outputs predictive image data generated inthe optimum intra prediction mode, prediction mode information on theoptimum intra prediction mode, and the cost function value in theoptimum intra prediction mode to the predictive image and optimum modeselecting section 33. In addition, to obtain amounts of generated codewhich amounts are used in calculation of the cost function values aswill be described later, the intra prediction section 31 outputs, in theintra prediction process in each intra prediction mode, the predictionmode information on the intra prediction mode to the lossless codingsection 16.

The motion prediction and compensation section 32 performs a motionprediction and compensation process in all prediction block sizescorresponding to macroblocks. The motion prediction and compensationsection 32 detects a motion vector for each image in each predictionblock size in the coding object image read from the picturerearrangement buffer 12 using the decoded image data after the filteringwhich decoded image data is read from the frame memory 25. Further, themotion prediction and compensation section 32 generates a predictiveimage by applying a motion compensation process to the decoded image onthe basis of the detected motion vector. In addition, the motionprediction and compensation section 32 calculates a cost function valuefor each prediction block size, and selects a prediction block size inwhich the calculated cost function value is a minimum, that is, aprediction block size in which best coding efficiency is obtained as anoptimum inter prediction mode. The motion prediction and compensationsection 32 outputs predictive image data generated in the optimum interprediction mode, prediction mode information on the optimum interprediction mode, and the cost function value in the optimum interprediction mode to the predictive image and optimum mode selectingsection 33. In addition, to obtain amounts of generated code whichamounts are used in calculation of the cost function values, the motionprediction and compensation section 32 outputs, in an inter predictionprocess in each prediction block size, the prediction mode informationon the inter prediction mode to the lossless coding section 16.Incidentally, the motion prediction and compensation section 32 alsoperforms prediction in a skipped macroblock and a direct mode as aninter prediction mode.

The predictive image and optimum mode selecting section 33 compares thecost function value supplied from the intra prediction section 31 withthe cost function value supplied from the motion prediction andcompensation section 32 in a macroblock unit, and selects the smallercost function value as an optimum mode in which best coding efficiencyis obtained. In addition, the predictive image and optimum modeselecting section 33 outputs the predictive image data generated in theoptimum mode to the subtracting section 13 and the adding section 23.Further, the predictive image and optimum mode selecting section 33outputs the prediction mode information on the optimum mode to thelossless coding section 16 and the filter setting section 41.Incidentally, the predictive image and optimum mode selecting section 33performs intra prediction or inter prediction in slice units.

The filter setting section 41 generates the parameter value for settingthe tap length of the filter and the filtering object pixel rangeaccording to the prediction block size indicated by the prediction modeinformation on the optimum mode, and outputs the parameter value to thedeblocking filter 24.

<2. Filtering of Deblocking Filter>

The coding system of H264./AVC allows the following three ways offiltering of the deblocking filter to be specified by two parameters ofdeblocking_filter_control_present_flag of a Picture Parameter Set RBSPincluded in image compression information anddisable_deblocking_filter_idc included in a Slice Header.

(a) Applied to a block boundary and a Macroblock boundary

(b) Applied only to a Macroblock boundary

(c) Not applied

As for the quantization parameter QP, QPY is used when the followingprocess is applied to luminance data, and QPC is used when the followingprocess is applied to color-difference data. In addition, in motionvector coding, intra prediction, and entropy coding (CAVLC/CABAC), apixel value belonging to a different slice is treated as “notavailable.” Further, in filtering, even a pixel value belonging to adifferent slice is treated as “available” when belonging to a samepicture.

In the following description, suppose that as shown in FIG. 2(A), pixeldata in blocks P and Q adjacent to each other before filtering at ablock boundary is p0 to p4 and q0 to q4 from the position of theboundary. In addition, suppose that as shown in FIG. 2(B), pixel dataafter the filtering is p0′ to p4′ and q0′ to q4′ from the position ofthe boundary.

Prior to the filtering, as shown in Table 1, block boundary strengthdata Bs (Boundary Strength) is defined for a pixel p and a pixel q inFIG. 2.

TABLE 1 One of p and q belongs to an intra Bs = 4 macroblock, and islocated at a Macroblock (Strongest boundary. Filtering) One of p and qbelongs to an intra Bs = 3 macroblock, but is not located at aMacroblock boundary. Neither of p and q belongs to an intra Bs = 2macroblock, and one of p and q has a transform coefficient. Neither of pand q belongs to an intra Bs = 1 macroblock, nor has a transformcoefficient. However, reference frames are different, the numbers ofreference frames are different, or mv values are different. Neither of pand q belongs to an intra Bs = 0 macroblock, nor has a transform (Nocoefficient. Reference frames and mv Filtering) values are the same.

As shown in Table 1, the block boundary strength data Bs is assigned ahighest filter strength of “4” when one of the pixel p and the pixel qbelongs to an intra coded macroblock MB, and the pixel in question islocated at the boundary of the macroblock MB.

The block boundary strength data Bs is assigned “3,” which is a highfilter strength next to “4,” when one of the pixel p and the pixel qbelongs to an intra coded macroblock MB, and the pixel in question isnot located at the boundary of the macroblock MB.

The block boundary strength data Bs is assigned “2,” which is a highfilter strength next to “3,” when neither of the pixel p and the pixel qbelongs to an intra coded macroblock MB, and one of the pixels has atransform coefficient.

The block boundary strength data Bs is assigned “1” when a conditionthat neither of the pixel p and the pixel q belongs to an intra codedmacroblock MB, and one of the pixels has no transform coefficient issatisfied, and a condition that reference frames are different, thenumbers of reference frames are different, or motion vectors aredifferent is satisfied.

The block boundary strength data Bs is assigned “0” when neither of thepixel p and the pixel q belongs to an intra coded macroblock MB, nor hasa transform coefficient, and reference frames and motion vectors are thesame. Incidentally, “0” indicates that the filtering is not performed.

(p2, p1, p0, q0, q1, q2) in FIG. 2 is filtered only when the conditionof Equation (1) holds.

Bs>0

|p0−q0|<α:|p1−p0|<β:|q1−q0|<β  (1)

Threshold values α and β as a parameter value for adjusting filterstrength, that is, a degree of ease of filtering are determined bydefault according to the quantization parameter QP as follows. Inaddition, a user can adjust the strength by two parameters ofslice_alpha_c0_offset_div2 and slice_beta_offset_div2 included in aSlice Header in image compression information. Incidentally, FIG. 3shows relation between the quantization parameter QP and the thresholdvalue α. When an amount of offset is added to the quantization parameterQP, a curve indicating the relation between the quantization parameterQP and the threshold value α moves in a direction of an arrow. It isthus clear that the filter strength is adjusted.

In addition, indexA and indexB are calculated from Equations (2) to (4)using the respective quantization parameters qPp and qPq of the block Pand the block Q adjacent to each other, and the threshold values α and βare obtained from tables shown in Table 2.

qPav=(qPp+qPq+1)>>1  (2)

indexA=Clip3(0,51,qPav+FilterOffsetA)  (3)

indexB=Clip3(0,51,qPav+FilterOffsetB)  (4)

TABLE 2 indexA (for α) or indexB (for β) 0 1 2 3 4 5 6 7 8 9 10 11 12 1314 15 16 17 18 α 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 4 5 β 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 2 2 2 indexA (for α) or indexB (for β) 19 20 21 22 23 2425 26 27 28 29 30 31 32 33 34 35 36 37 α 6 7 8 9 10 12 13 15 17 20 22 2528 32 36 40 45 50 56 β 3 3 3 3 4 4 4 6 6 7 7 8 8 9 9 10 10 11 11 indexA(for α) or indexB (for β) 38 39 40 41 42 43 44 45 46 47 48 49 50 51 α 6371 80 90 101 113 127 144 162 182 203 226 255 255 β 12 12 13 13 14 14 1515 16 16 17 17 18 18

Different methods of filtering are defined for a case where “Bs<4” and acase where “Bs=4.”

Description will first be made of the case where “Bs<4.”

The deblocking filter performs operation shown in Equations (5) to (7)to calculate the pixel data p0′ and q0′ after the filtering.

Clip3 in Equation (7) denotes a clipping process.

p0′=Clip1(p0+Δ)  (5)

q0′=Clip1(q0+Δ)  (6)

Δ=Clip3(−tc,tc((((q0−p0)<<2)+(p1−q1)+4)>>3))  (7)

The deblocking filter calculates “tc” in Equation (7) on the basis ofEquation (8) when chromaEdgeFlag indicates “0,” and otherwise calculates“tc” in Equation (7) on the basis of Equation (9).

In Equation (8), “( )1:0” indicates “1” when a condition within ( ) issatisfied, and otherwise indicates “0.”

tc=tc0+((ap<β)?1:0)+aq<β)?1:0)  (8)

tc=tc0+1  (9)

The value of tc is defined as in Table 3 according to the values of Bsand indexA.

TABLE 3 indexA 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 2122 23 24 25 bS = 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1bS = 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 bS = 3 0 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 indexA 26 27 28 29 30 3132 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 bS = 1 1 1 11 1 1 1 2 2 2 2 3 3 3 4 4 4 5 6 6 7 8 9 10 11 13 bS = 2 1 1 1 1 1 2 2 22 3 3 3 4 4 5 5 6 7 8 8 10 11 12 13 15 17 bS = 3 1 2 2 2 2 3 3 3 4 4 4 56 6 7 8 9 10 11 13 14 16 18 20 23 25

In addition, the deblocking filter calculates ap and aq in Equation (8)according to Equations (10) and (11).

ap=|p2−p0|  (10)

aq=|q2−q0|  (11)

The deblocking filter calculates the pixel data p1′ after the filteringby performing operation shown in Equation (12) when chromaEdgeFlag is“0” and ap is equal to or less than “β,” and otherwise obtains the pixeldata p1′ after the filtering by Equation (13).

p1′=p1+Clip3(−tc0,tc0,(p2+((p0+q0+1)>>1)−(p1<<1))>>1)  (12)

p1′=p1  (13)

The deblocking filter calculates the pixel data q1′ after the filteringby performing operation shown in Equation (14) when chromaEdgeFlag is“0” and aq is equal to or less than “β,” and otherwise obtains the pixeldata q1′ after the filtering by Equation (15).

q1′=q1+Clip3(−tc0,tc0,(q2+((p0+q0+1)>>1)−(q1<<1))>>1)  (14)

q1′=q1  (15)

In addition, the pixel data p2′ and the pixel data q2′ are values beforethe filter.

p2′=p2  (16)

q2′q2  (17)

Description will next be made of the case where “Bs=4.”

The deblocking filter calculates the pixel data p0′, p1′, and p2′according to Equations (19) to (21) when chromaEdgeFlag indicates “0”and the condition of Equation (18) is satisfied.

ap<β&&|p0−q0|<((α>>2)+2)  (18)

p0′=(p2+2·p1+2·p0+2·q0+q1+4)>>3  (19)

p1′(p2+p1+p0+q0+2)>>2  (20)

p2′=(2·p3+3·p2+p1+p0+q0+4)>>3  (21)

The deblocking filter calculates the pixel data p0′, p1′, and p2′according to Equations (22) to (24) when chromaEdgeFlag indicates “0”and the condition of Equation (18) is not satisfied.

p0′=(2·p1+p0+q1+2)>>2  (22)

p1′=p1  (23)

p2′=p2  (24)

The deblocking filter calculates the pixel data q0′, q1′, and q2′according to Equations (26) to (28) when chromaEdgeFlag indicates “0”and the condition of Equation (25) is satisfied.

aq<β&&|p0−q0|<((α>>2)+2)  (25)

q0′=(p1+2·p0+2·q0+2·q1+q2+4)>>3  (26)

q1′=(p0+q0+q1+q2+2)>>2  (27)

q2′=(2·q3+3·q2+q1+q0+p4+4)>>3  (28)

The deblocking filter calculates the pixel data q0′, q1′, and q2′according to Equations (29) to (31) when chromaEdgeFlag indicates “0”and the condition of Equation (25) is not satisfied.

q0′=(2·q1+q0+p1+2)>>2  (29)

q1′=q1  (30)

q2′=q2  (31)

Thus, in the coding system of H264./AVC, the pixel data p0′ to p2′ andq0′ to q2′ is calculated by performing filtering using the pixel data p0to p3 and q0 to q3.

<3. Configuration of Deblocking Filter and Filter Setting Section inImage Coding Device>

The filter setting section 41 sets a tap length and a filtering objectpixel range in the deblocking filter 24 according to the predictionblock size of the optimum mode in the macroblock in question.

In general, block distortion is conspicuous to the human eye in a caseof a larger block size. In addition, a larger block size tends to beselected for a flat region not including much texture information.

Thus, the filter setting section 41 performs case classificationaccording to the block sizes on adjacent sides of two blocks adjacent toeach other, and sets the tap length of filtering and the filteringobject pixel range according to a result of the case classification. Thefilter setting section 41 performs the case classification into forexample a case where the adjacent sides of the two blocks adjacent toeach other both have a predetermined block size or smaller and a casewhere at least one of the two blocks adjacent to each other has anextended block size larger than the predetermined block size.

In the case where the block sizes on the adjacent sides of the twoblocks adjacent to each other are both a predetermined block size orsmaller, for example a macroblock size of the H.264/AVC standard, thefilter setting section 41 calculates the pixel data p0′ to p2′ and q0′to q2′ by performing filtering as described above. In the case where atleast one of the two blocks adjacent to each other has an extended blocksize larger than the predetermined block size, the filter settingsection 41 extends the tap length or the filtering object pixel rangeaccording to the block size of the block boundary. By extending the taplength or the filtering object pixel range, the filter setting section41 subjects a part of a large block size with a conspicuous blockdistortion to a smoothing process of higher strength and filtering thatis applied to even values of pixels more distant from the blockboundary. Thus, the block distortion is made less conspicuous, and thesubjective image quality of the decoded image is made more desirable.

Incidentally, the high-frequency component of the image is lost when asmoothing process of higher strength is performed. However, a largeblock size is often applied to a relatively flat region with a smallhigh-frequency component within the image, so that subjectivedegradation such as a loss of texture or the like does not occur.

The filter setting section 41 thus generates parameter values indicatingthe tap length and the filtering object pixel range set according to theprediction block size, and supplies the parameter values to thedeblocking filter 24. In addition, when macroblocks of a plurality ofdifferent sizes larger than the block size of a predetermined macroblockare used, the filter setting section 41 may set the tap length longerand the filtering object pixel range wider as the block size isincreased.

FIG. 4 shows a configuration of the deblocking filter and the filtersetting section. The filter setting section 41 includes a block sizebuffer 411 and a parameter value generating portion 412. In addition,the deblocking filter 24 includes a filter strength determining portion241 and a filtering portion 242.

The block size buffer 411 accumulates information for one frame imagewhich information indicates the prediction block size in the optimummode selected by the predictive image and optimum mode selecting section33. That is, the block size buffer 411 stores information on theprediction block size of each macroblock in one frame image as a codingobject.

The parameter value generating portion 412 determines the predictionblock sizes on adjacent sides of two blocks adjacent to each other onthe basis of the prediction block size information of the block sizebuffer 411. The parameter value generating portion 412 generatesparameter values for setting the tap length of filtering and thefiltering object pixel range for a block boundary between the two blockson the basis of the determined prediction block sizes, and supplies theparameter values to the filter strength determining portion 241.

The filter strength determining portion 241 determines block boundarystrength data Bs on the basis of the prediction mode informationsupplied from the lossless coding section 16, and outputs the determinedblock boundary strength data Bs and the parameter values supplied fromthe parameter value generating portion 412 to the filtering portion 242.

The filtering portion 242 performs filtering with the block boundarystrength data Bs as well as the tap length and the filtering objectpixel range indicated by the parameter values to calculate pixel dataafter the filtering.

An illustration will be given in the following of case classificationinto a first case where the prediction block sizes on the adjacent sidesof the two blocks adjacent to each other are both a predetermined blocksize (16×16 pixels) or smaller and a second case where at least one ofthe two blocks adjacent to each other has an extended block size largerthan the predetermined block size. In this case, the filtering of theH.264/AVC coding system described above is performed in the first case.In addition, in the second case, the tap length is set to an extendedlength to increase the strength of smoothing, and/or the filteringobject pixel range is extended to perform filtering up to pixels atpositions distant from the block boundary.

An illustration will next be given of filtering when the tap length andthe filtering object pixel range are extended.

The filtering portion 242 extends the tap length and the filteringobject pixel range on the basis of the parameter values, performsfiltering, and calculates pixel data p0′ to p3′ and q0′ to q3′ after thefiltering from pixel data p0 to p4 and q0 to q4. In this case, thefiltering portion 242 uses Equation (32) in place of the above Equation(7).

Δ=Clip3(−tc,tc((((q0−p0)<<3)+((p1−q1)<<1)+(p2−q2)+8)>>4))  (32)

Further, the filtering portion 242 calculates the pixel data p1′ and q1′by using Equations (33) and (34) in place of Equations (12) and (14).

p1′=p1+Clip3(−tc0,tc0),(p3+p2+p0+((q0+q1+1)>>1)(p1<<2))>>2)  (33)

q1′=q1+Clip3(−tc0,tc0,(q3+q2+q0+((q0+q1+1)>>1)(q1<<2))>>2)  (34)

In addition, the filtering portion 242 calculates the pixel data p2′ andq2′ by using Equations (35) and (36) in place of Equations (16) and(17).

p2′=p2+Clip3(−tc0,tc0,(p4+p3+p1+((p0+q1+1)>>1)(p2<<2))>>2)  (35)

q2′=q2+Clip3(−tc0,tc0,(q4+q3+q1+((q0+q1+1)>>1)(q2<<2))>>2)  (36)

In addition, when chromaEdgeFlag indicates “0” and the condition ofEquation (18) is satisfied, the filtering portion 242 calculates thepixel data p0′, p1′, p2′, and p3′ according to Equations (37) to (40).

p0′=(p3+2·p2+3·p1+4·p0+3·q0+2·q1+q2+8)>>4  (37)

p1′=(p3+p2+2·p1+2·p0+q0+q1+4)>>3  (38)

p2′=(p4+3·p3+4·p2+3·p1+2·p0+2·q0+q1+8)>>4  (39)

p3′=(p4+3·p3+p2+p1+p0+q0+4)>>3  (40)

In addition, when chromaEdgeFlag indicates “0” and the condition ofEquation (25) is satisfied, the filtering portion 242 calculates thepixel data q0′, q1′, q2′, and q3′ according to Equations (41) to (44).

q0′=(p2+2·p1+3·p0+4·q0+3·q1+2·q2+q3+8)>>4  (41)

q1′=(p1+p0+2·q0+2·q1+q2+q3+4)>>3  (42)

q2′=(q4+3·q3+4·q2+3·q1+1·q0+2·p0+p1+8)>>4  (43)

q3′=(q4+3·q3+q2+q1+q0+p0+4)>>3  (44)

Incidentally, the setting of the tap length and the filtering objectpixel range is not limited to case classification into two cases, thatis, a case where two blocks both have a size of 16×16 pixels or asmaller size and a case where at least one of the two blocks is largerthan 16×16 pixels. For example, case classification may be performedinto a case where two blocks both have a size of 16×16 pixels or asmaller size, a case where at least one of the two blocks is larger than16×16 pixels and both of the two blocks are 32×32 pixels or smaller, anda case where at least one of the two blocks is larger than 32×32 pixels.In this case, at a boundary of a larger block size, the tap length isfurther lengthened to increase the strength of smoothing, and thefiltering object pixel range is further widened to filter values ofpixels distant from the block boundary. In addition, in the setting ofthe tap length and the filtering object pixel range, only one of the taplength and the filtering object pixel range may be extended according toa result of case classification.

Thus, the image coding device sets the tap length of the filter and thefiltering object pixel range according to the block sizes on theadjacent sides of two blocks adjacent to each other, and subjects a partof a large block size with a conspicuous block distortion to smoothingof higher strength and filtering that is applied to even values ofpixels more distant from the block boundary. Therefore, the blockdistortion is made less conspicuous, and the image quality of thedecoded image used to generate a predictive image can be made moredesirable.

<4. Operation of Image Coding Device>

An image coding process operation will next be described. FIG. 5 showsprediction block sizes used in an image coding process. The H.264/AVCsystem defines prediction block sizes of 16×16 pixels and 4×4 pixels asshown in FIGS. 5(C) and 5(D). In addition, when macroblocks of anextended size larger than that of the H.264/AVC system are used, forexample when macroblocks of 32×32 pixels are used, prediction blocksizes shown in FIG. 5(B), for example, are defined. When macroblocks of64×64 pixels are used, prediction block sizes shown in FIG. 5(A), forexample, are defined.

Incidentally, “Skip/direct” in FIG. 5 denotes a prediction block sizewhen a skipped macroblock or a direct mode is selected in the motionprediction and compensation section 32. In addition, “ME” denotes amotion compensation block size. In addition, “P8×8” denotes that furtherdivision can be made in a lower layer in which the size of themacroblock is reduced.

FIG. 6 is a flowchart of an image coding process operation. In stepST11, the A/D converting section 11 subjects an input image signal toA/D conversion.

In step ST12, the picture rearrangement buffer 12 performs picturerearrangement. The picture rearrangement buffer 12 stores the image datasupplied from the A/D converting section 11, and performs rearrangementfrom order of display of each picture to order of coding of eachpicture.

In step ST13, the subtracting section 13 generates prediction errordata. The subtracting section 13 generates the prediction error data bycalculating differences between the image data of an image rearranged instep ST12 and predictive image data selected by the predictive image andoptimum mode selecting section 33. The prediction error data is reducedin amount as compared with the original image data. Therefore, theamount of data can be compressed as compared with a case where the imageis coded as it is. Incidentally, when the predictive image and optimummode selecting section 33 selects a predictive image supplied from theintra prediction section 31 and a predictive image from the motionprediction and compensation section 32 in slice units, intra predictionis performed in a slice for which the predictive image supplied from theintra prediction section 31 is selected. In addition, inter predictionis performed in a slice for which the predictive image from the motionprediction and compensation section 32 is selected.

In step ST14, the orthogonal transform section 14 performs an orthogonaltransform process. The orthogonal transform section 14 subjects theprediction error data supplied from the subtracting section 13 to anorthogonal transform. Specifically, the prediction error data issubjected to an orthogonal transform such as a Discrete CosineTransform, a Karhunen-Loeve transform, or the like, and transformcoefficient data is output.

In step ST15, the quantizing section 15 performs a quantizing process.The quantizing section 15 quantizes the transform coefficient data. Inthe quantization, rate control is performed, as will be described laterin description of the process of step ST25.

In step ST16, the dequantizing section 21 performs a dequantizingprocess. The dequantizing section 21 dequantizes the transformcoefficient data quantized by the quantizing section 15 withcharacteristics corresponding to the characteristics of the quantizingsection 15.

In step ST17, the inverse orthogonal transform section 22 performs aninverse orthogonal transform process. The inverse orthogonal transformsection 22 subjects the transform coefficient data dequantized by thedequantizing section 21 to an inverse orthogonal transform withcharacteristics corresponding to the characteristics of the orthogonaltransform section 14.

In step ST18, the adding section 23 generates decoded image data. Theadding section 23 generates the decoded image data by adding togetherthe predictive image data supplied from the predictive image and optimummode selecting section 33 and the data after the inverse orthogonaltransform at a position corresponding to the predictive image.

In step ST19, the deblocking filter 24 performs filtering. Thedeblocking filter 24 removes block distortion by filtering the decodedimage data output from the adding section 23.

In step ST20, the frame memory 25 stores the decoded image data. Theframe memory 25 stores the decoded image data before the filtering andthe decoded image data after the filtering.

In step ST21, the intra prediction section 31 and the motion predictionand compensation section 32 each perform a predicting process.Specifically, the intra prediction section 31 performs an intraprediction process in an intra prediction mode, and the motionprediction and compensation section 32 performs a motion prediction andcompensation process in an inter prediction mode. Details of thepredicting process will be described later in detail with reference toFIG. 7. This process performs respective predicting processes in allprediction modes as candidates, and calculates respective cost functionvalues of all the prediction modes as candidates. Then, an optimum intraprediction mode and an optimum inter prediction mode are selected on thebasis of the calculated cost function values, and predictive imagesgenerated in the selected prediction modes, the cost functions thereof,and prediction mode information are supplied to the predictive image andoptimum mode selecting section 33.

In step ST22, the predictive image and optimum mode selecting section 33selects predictive image data. The predictive image and optimum modeselecting section 33 determines an optimum mode in which best codingefficiency is obtained on the basis of the respective cost functionvalues output from the intra prediction section 31 and the motionprediction and compensation section 32. Further, the predictive imageand optimum mode selecting section 33 selects predictive image data inthe determined optimum mode, and supplies the predictive image data tothe subtracting section 13 and the adding section 23. This predictiveimage is used in the operations of steps ST13 and ST18, as describedabove. Incidentally, the prediction mode information corresponding tothe selected predictive image data is output to the lossless codingsection 16 and the filter setting section 41.

In step ST23, the lossless coding section 16 performs a lossless codingprocess. The lossless coding section 16 losslessly codes the quantizeddata output from the quantizing section 15. Specifically, the quantizeddata is subjected to lossless coding such as variable-length coding,arithmetic coding, or the like, and the data is compressed. At thistime, the prediction mode information (including for example amacroblock type, a prediction mode, motion vector information, referencepicture information, and the like) input to the lossless coding section16 in step ST22 described above and the like are also losslessly coded.Further, the losslessly coded data of the prediction mode information isadded to the header information of a coded stream generated bylosslessly coding the quantized data.

In step ST24, the storage buffer 17 performs a storing process to storethe coded stream. The coded stream stored in the storage buffer 17 isread out as appropriate, and transmitted to a decoding side via atransmission line.

In step ST25, the rate controlling section 18 performs rate control. Therate controlling section 18 controls the rate of quantizing operation ofthe quantizing section 15 so as to prevent the storage buffer 17 from anoverflow or an underflow when the storage buffer 17 stores the codedstream.

The predicting process in step ST21 in FIG. 6 will next be describedwith reference to a flowchart of FIG. 7.

In step ST31, the intra prediction section 31 performs an intraprediction process. The intra prediction section 31 performs intraprediction of an image of a block as a processing object in all intraprediction modes as candidates. Incidentally, the decoded image datastored in the frame memory 25 without being filtered by the deblockingfilter 24 is used as image data of a decoded image referred to in theintra prediction. Details of the intra prediction process will bedescribed later. This process performs intra prediction in all the intraprediction modes as candidates, and calculates cost function values forall the intra prediction modes as candidates. Then, one intra predictionmode in which best coding efficiency is obtained is selected from allthe intra prediction modes on the basis of the calculated cost functionvalues.

In step ST32, the motion prediction and compensation section 32 performsan inter prediction process. The motion prediction and compensationsection 32 performs the inter prediction process in all inter predictionmodes (all prediction block sizes) as candidates using the decoded imagedata after the filtering which decoded image data is stored in the framememory 25. Details of the inter prediction process will be describedlater. This process performs prediction processes in all the interprediction modes as candidates, and calculates cost function values forall the inter prediction modes as candidates. Then, one inter predictionmode in which best coding efficiency is obtained is selected from allthe inter prediction modes on the basis of the calculated cost functionvalues.

The intra prediction process in step ST31 in FIG. 7 will next bedescribed with reference to a flowchart of FIG. 8.

In step ST41, the intra prediction section 31 performs intra predictionin each prediction mode. The intra prediction section 31 generatespredictive image data for each intra prediction mode using the decodedimage data before the filtering which decoded image data is stored inthe frame memory 25.

In step ST42, the intra prediction section 31 calculates a cost functionvalue for each prediction mode. The cost function value is calculated onthe basis of the method of one of a High Complexity mode and a LowComplexity mode as defined in a JM (Joint Model) as reference softwarein the H.264/AVC system.

Specifically, in the High Complexity mode, as the process of step ST41,up to the lossless coding process is tentatively performed for all theprediction modes as candidates, and a cost function value expressed bythe following Equation (45) is calculated for each prediction mode.

Cost(ModeεΩ)=D+λ·R  (45)

Ω denotes a universal set of the prediction modes as candidates forcoding a block or a macroblock in question. D denotes difference energy(distortion) between a decoded image and an input image when coding isperformed in a prediction mode. R is an amount of code generatedincluding orthogonal transform coefficients, prediction modeinformation, and the like. λ is a Lagrange multiplier given as afunction of the quantization parameter QP.

That is, coding in the High Complexity Mode requires a tentativeencoding process to be performed once in all the prediction modes ascandidates to calculate the above parameters D and R, and needs a largeramount of operation.

On the other hand, in the Low Complexity mode, as the process of stepST41, a predictive image is generated and up to header bits of motionvector information, prediction mode information, and the like arecalculated for all the prediction modes as candidates, and a costfunction value expressed by the following Equation (46) is calculatedfor each prediction mode.

Cost(ModeεΩ)=D+QPtoQuant(QP)·Header_Bit  (46)

Ω denotes a universal set of the prediction modes as candidates forcoding a block or a macroblock in question. D denotes difference energy(distortion) between a decoded image and an input image when coding isperformed in a prediction mode. Header_Bit is header bits for theprediction mode. QPtoQuant is a function given as a function of thequantization parameter QP.

That is, the Low Complexity Mode requires the prediction process to beperformed for each prediction mode, but does not require a decodedimage, so that the prediction process can be realized with a smalleramount of operation than in the High Complexity Mode.

In step ST43, the intra prediction section 31 determines an optimumintra prediction mode. On the basis of the cost function valuescalculated in step ST42, the intra prediction section 31 selects oneintra prediction mode whose cost function value is a minimum value amongthe cost function values, and determines the intra prediction mode asthe optimum intra prediction mode.

The inter prediction process in step ST32 in FIG. 7 will next bedescribed with reference to a flowchart of FIG. 9.

In step ST51, the motion prediction and compensation section 32determines a motion vector and a reference image for each predictionmode. That is, the motion prediction and compensation section 32determines a motion vector and a reference image for a block as aprocessing object in each prediction mode.

In step ST52, the motion prediction and compensation section 32 performsmotion compensation for each prediction mode. The motion prediction andcompensation section 32 applies motion compensation to the referenceimage in each prediction mode (each prediction block size) on the basisof the motion vector determined in step ST51, and generates predictiveimage data for each prediction mode.

In step ST53, the motion prediction and compensation section 32generates motion vector information for each prediction mode. The motionprediction and compensation section 32 generates motion vectorinformation to be included in the coded stream which information isabout the motion vector determined in each prediction mode. For example,a predicted motion vector is determined using median prediction or thelike, and motion vector information indicating a difference between themotion vector detected by motion prediction and the predicted motionvector is generated. The thus generated motion vector information isalso used to calculate a cost function value in next step ST54, and isincluded in prediction mode information and output to the losslesscoding section 16 when the predictive image and optimum mode selectingsection 33 finally selects the corresponding predictive image.

In step ST54, the motion prediction and compensation section 32calculates a cost function value for each inter prediction mode. Themotion prediction and compensation section 32 calculates the costfunction value using Equation (45) or Equation (46) described above.Incidentally, the calculation of cost function values for interprediction modes includes evaluation of cost function values in the SkipMode and the Direct Mode defined in the H.264/AVC system.

In step ST55, the motion prediction and compensation section 32determines an optimum inter prediction mode. On the basis of the costfunction values calculated in step ST54, the motion prediction andcompensation section 32 selects one prediction mode whose cost functionvalue is a minimum value among the cost function values, and determinesthe prediction mode as the optimum inter prediction mode.

A filter setting process will next be described with reference to aflowchart of FIG. 10. Incidentally, FIG. 10 represents a case where thetap length and the filtering object pixel range are extended.

In step ST61, the filter setting section 41 obtains the prediction blocksize in the optimum mode. The filter setting section 41 obtains theprediction block size corresponding to the predictive image selected instep ST22 in FIG. 6, that is, the prediction block size when coding isperformed in the optimum mode.

In step ST62, the filter setting section 41 determines whether a blockin question or an adjacent block is larger than 16×16 pixels. When atleast one of the block in question and the adjacent block is larger than16×16 pixels, the filter setting section 41 proceeds to step ST63. Whenthe block in question and the adjacent block are both 16×16 pixels orsmaller, the filter setting section 41 proceeds to step ST64.

In step ST63, the filter setting section 41 extends and sets the taplength and the filtering object pixel range. For example, the filtersetting section 41 extends the tap length and the filtering object pixelrange more than in the H.264/AVC coding system, and makes pixel data p0′to p3′ and q0′ to q3′ after filtering calculated as described above.

In step ST64, the filter setting section 41 sets the tap length and thefiltering object pixel range without extension. For example, the filtersetting section 41 sets the tap length and the filtering object pixelrange of the H.264/AVC system, and makes pixel data p0′ to p2′ and q0′to q2′ after filtering calculated as described above.

Thus, according to the image coding device and the image coding methodto which the present technology is applied, a prediction block sizeproviding best coding efficiency is determined, and image data is codedin the determined prediction block size. At this time, informationindicating the prediction block size is stored in the block size buffer411 in the filter setting section 41. Therefore, the position of theprediction block in a decoded image is clear when decoded image data isgenerated by decoding the image data coded in the prediction block sizeproviding the best coding efficiency. Thus, the tap length and thefiltering object pixel range are set according to the prediction blocksize on the basis of the information stored in the block size buffer411, whereby block distortion can be reduced even when the predictionblock size is large. In addition, because the block distortion in thedecoded image data for generating a predictive image can be reduced, anincrease in amount of prediction error data due to effect of the blockdistortion can be prevented. Thus, an amount of data after the codingprocess can be further reduced.

<5. Configuration of Image Decoding Device>

The coded stream generated by coding an input image is supplied to animage decoding device via a predetermined transmission line, a recordingmedium, or the like, and decoded.

FIG. 11 shows a configuration of an image decoding device. The imagedecoding device 50 includes a storage buffer 51, a lossless decodingsection 52, a dequantizing section 53, an inverse orthogonal transformsection 54, an adding section 55, a deblocking filter 56, a picturerearrangement buffer 57, and a D/A converting section 58. The imagedecoding device 50 further includes a frame memory 61, selectors 62 and65, an intra prediction section 63, a motion compensation section 64,and a filter setting section 71.

The storage buffer 51 stores the transmitted coded stream. The losslessdecoding section 52 decodes the coded stream supplied from the storagebuffer 51 by a system corresponding to the coding system of the losslesscoding section 16 in FIG. 1. In addition, the lossless decoding section52 outputs prediction mode information obtained by decoding the headerinformation of the coded stream to the intra prediction section 63, themotion compensation section 64, and the deblocking filter 56.

The dequantizing section 53 dequantizes the quantized data decoded bythe lossless decoding section 52 by a system corresponding to thequantizing system of the quantizing section 15 in FIG. 1. The inverseorthogonal transform section 54 subjects the output of the dequantizingsection 53 to an inverse orthogonal transform by a system correspondingto the orthogonal transform system of the orthogonal transform section14 in FIG. 1, and outputs the data after the inverse orthogonaltransform to the adding section 55.

The adding section 55 generates decoded image data by adding togetherthe data after the inverse orthogonal transform and predictive imagedata supplied from the selector 65, and outputs the decoded image datato the deblocking filter 56 and the frame memory 61.

The deblocking filter 56 is configured in a similar manner to thedeblocking filter 24 in FIG. 1. The deblocking filter 56 removes blockdistortion by filtering the decoded image data supplied from the addingsection 55. Then, the deblocking filter 56 supplies the decoded imagedata to the frame memory 61 to make the decoded image data stored in theframe memory 61, and outputs the decoded image data to the picturerearrangement buffer 57. In addition, the deblocking filter 56 sets atap length and a filtering object pixel range to perform the filtering,on the basis of the prediction mode information supplied from thelossless decoding section 52 and parameter values supplied from thefilter setting section 71 to be described later.

The picture rearrangement buffer 57 performs picture rearrangement.Specifically, frames rearranged for coding by the picture rearrangementbuffer 12 of FIG. 1 are rearranged into the original display order, andthe rearranged image data is output to the D/A converting section 58.

The D/A converting section 58 subjects the image data supplied from thepicture rearrangement buffer 57 to D/A conversion, and makes the imagedisplayed by outputting the image data to a display not shown in thefigure.

The frame memory 61 retains the decoded image data before the filteringwhich decoded image data is supplied from the adding section 55 and thedecoded image data after the filtering which decoded image data issupplied from the deblocking filter 24.

The selector 62 supplies the decoded image data before the filteringwhich decoded image data is read out from the frame memory 61 to theintra prediction section 63 when a prediction block in which intraprediction has been performed is decoded on the basis of the predictionmode information supplied from the lossless decoding section 52. Inaddition, the selector 26 supplies the decoded image data after thefiltering which decoded image data is read out from the frame memory 61to the motion compensation section 64 when a prediction block in whichinter prediction has been performed is decoded on the basis of theprediction mode information supplied from the lossless decoding section52.

The intra prediction section 63 generates a predictive image on thebasis of the prediction mode information supplied from the losslessdecoding section 52, and outputs the generated predictive image data tothe selector 65. In addition, the intra prediction section 63 outputsinformation indicating the block size of the generated predictive imageto the filter setting section 71.

The motion compensation section 64 performs motion compensation on thebasis of the prediction mode information supplied from the losslessdecoding section 52, generates predictive image data, and outputs thepredictive image data to the selector 65. That is, on the basis ofmotion vector information and reference frame information included inthe prediction mode information, the motion compensation section 64applies motion compensation to a reference image indicated by thereference frame information with a motion vector based on the motionvector information, and generates the predictive image data. Inaddition, the motion compensation section 64 outputs informationindicating the block size of the generated predictive image to thefilter setting section 71.

The selector 65 supplies the predictive image data generated in theintra prediction section 63 to the adding section 55. In addition, theselector 65 supplies the predictive image data generated in the motioncompensation section 64 to the adding section 55.

The filter setting section 71 is configured in a similar manner to thefilter setting section 41 shown in FIG. 4. In addition, the filtersetting section 71 stores information indicating the prediction blocksizes of decoded blocks. The filter setting section 71 sets the taplength and the filtering object pixel range according to the predictionblock sizes on adjacent sides of a block as a decoding object and adecoded block adjacent to the block as the decoding object. The filtersetting section 71 supplies parameter values indicating the set taplength and the set filtering object pixel range to the deblocking filter56. In addition, when the prediction block size in the optimum mode ofone of the block in question and the adjacent block is an extended blocksize, the filter setting section 71 sets the tap length and thefiltering object pixel range according to the larger prediction blocksize. In addition, when a plurality of macroblocks having a larger sizethan a predetermined macroblock are used, the filter setting section 71sets the tap length longer and the filtering object pixel range wider asthe size is increased.

<6. Operation of Image Decoding Device>

An image decoding process operation performed in the image decodingdevice 50 will next be described with reference to a flowchart of FIG.12.

In step ST71, the storage buffer 51 stores a transmitted coded stream.In step ST72, the lossless decoding section 52 performs a losslessdecoding process. The lossless decoding section 52 decodes the codedstream supplied from the storage buffer 51. That is, the quantized dataof each picture coded by the lossless coding section 16 in FIG. 1 isobtained. In addition, the lossless decoding section 52 losslesslydecodes prediction mode information included in the header informationof the coded stream, and supplies the obtained prediction modeinformation to the deblocking filter 56 and the selectors 62 and 65.Further, when the prediction mode information is information on an intraprediction mode, the lossless decoding section 52 outputs the predictionmode information to the intra prediction section 63. In addition, whenthe prediction mode information is information on an inter predictionmode, the lossless decoding section 52 outputs the prediction modeinformation to the motion compensation section 64.

In step ST73, the dequantizing section 53 performs a dequantizingprocess. The dequantizing section 53 dequantizes the quantized datadecoded by the lossless decoding section 52 with characteristicscorresponding to the characteristics of the quantizing section 15 inFIG. 1.

In step ST74, the inverse orthogonal transform section 54 performs aninverse orthogonal transform process. The inverse orthogonal transformsection 54 subjects the transform coefficient data dequantized by thedequantizing section 53 to an inverse orthogonal transform withcharacteristics corresponding to the characteristics of the orthogonaltransform section 14 in FIG. 1.

In step ST75, the adding section 55 generates decoded image data. Theadding section 55 generates the decoded image data by adding togetherthe data obtained by performing the inverse orthogonal transform processand predictive image data selected in step ST79 to be described later.The original image is thereby decoded.

In step ST76, the deblocking filter 56 performs filtering. Thedeblocking filter 56 filters the decoded image data output from theadding section 55 to remove block distortion included in the decodedimage.

In step ST77, the frame memory 61 stores the decoded image data.

In step ST78, the intra prediction section 63 and the motioncompensation section 64 generate predictive image data. The intraprediction section 63 and the motion compensation section 64 eachgenerate predictive image data so as to correspond to the predictionmode information supplied from the lossless decoding section 52.

Specifically, when the prediction mode information for intra predictionis supplied from the lossless decoding section 52, the intra predictionsection 63 performs an intra prediction process using the decoded imagedata in the frame memory 61 on the basis of the prediction modeinformation, and generates the predictive image data. In addition, whenthe prediction mode information for inter prediction is supplied fromthe lossless decoding section 52, the motion compensation section 64performs motion compensation using the decoded image data in the framememory 61 on the basis of the prediction mode information, and generatesthe predictive image data.

In step ST79, the selector 65 selects predictive image data.Specifically, the selector 65 selects the predictive image supplied fromthe intra prediction section 63 and the predictive image data generatedin the motion compensation section 64, and supplies the predictive imagedata to the adding section 55 to make the predictive image data added tothe output of the inverse orthogonal transform section 54 in step ST75as described above.

In step ST80, the picture rearrangement buffer 57 performs picturerearrangement. Specifically, the picture rearrangement buffer 57rearranges frames rearranged for coding by the picture rearrangementbuffer 12 in the image coding device 10 of FIG. 1 into the originaldisplay order.

In step ST81, the D/A converting section 58 subjects the image data fromthe picture rearrangement buffer 57 to D/A conversion. This image isoutput to a display not shown in the figures, and the image isdisplayed.

In addition, in the filtering in step ST76 in FIG. 12, the filtersetting section 71 performs the filter setting process shown in FIG. 10described above. The filter setting section 71 sets the tap length andthe filtering object pixel range according to the prediction block sizeson adjacent sides of a block as a decoding object and a decoded blockadjacent to the block as the decoding object. When the block size on theadjacent side of at least one of the block in question and the adjacentblock is extended, the filter setting section 71 sets an extended taplength and an extended filtering object pixel range. Incidentally, theprediction block size on the adjacent side of the decoded block isstored in a block size buffer in the filter setting section 71 when theadjacent block is decoded. The filter setting section 71 generatesparameter values indicating the set tap length and the set filteringobject pixel range, and outputs the parameter values to the deblockingfilter 56. The deblocking filter 56 applies the filtering to a blockboundary between the block as the decoding object and the decoded blockadjacent to the block as the decoding object with the tap length and thefiltering object pixel range indicated by the parameter values suppliedfrom the filter setting section 71.

Thus, according to the image decoding device and the image decodingmethod to which the present technology is applied, informationindicating prediction block size used in a coding process is stored inthe block size buffer in the filter setting section 71. Therefore, theposition of a prediction block in a decoded image is clear when thedecoded image data for image display is generated by decoding the codedstream. Thus, on the basis of the information stored in the block sizebuffer, a part of a large block size with a conspicuous block distortionis subjected to filtering that is applied to even values of pixels moredistant from the block boundary and a smoothing process of higherstrength. Hence, a decoded image of excellent image quality with aninconspicuous block distortion can be obtained.

Further, it suffices for the filter setting section of an imageprocessing device to set the tap length and the filtering object pixelrange according to whether the decoded image data is image data forgenerating a predictive image or image data for image display. When theimage processing device is an image coding device, for example, thefilter setting section makes settings for the decoded image data forgenerating a predictive image so that the image quality of the decodedimage used to generate the predictive image is an excellent imagequality and the amount of data of the coded stream is reduced. Inaddition, when the image processing device is an image decoding device,the filter setting section makes settings so that the image quality ofthe decoded image used for image display is an image quality desired bya user. Thus, it is possible to perform deblocking filtering suitablefor image coding when the filter setting section is provided in theimage coding device and deblocking filtering suitable for the imagedecoding device which deblocking filtering is performed by the filtersetting section.

The series of processes described in the specification can be performedby hardware, software, or a composite configuration of both hardware andsoftware. When processing is performed by software, a program in which aprocessing sequence is recorded is installed into a memory within acomputer incorporated in dedicated hardware, and executed.Alternatively, the program can be installed on a general-purposecomputer capable of performing various kinds of processing, andexecuted.

For example, the program can be recorded in advance on a hard disk as arecording medium or in a ROM (Read Only Memory) as a recording medium.Alternatively, the program can be stored (recorded) temporarily orpermanently on a removable recording medium such as a flexible disk, aCD-ROM (Compact Disc Read Only Memory), an MO (Magneto optical) disk, aDVD (Digital Versatile Disc), a magnetic disk, a semiconductor memory,or the like. Such a removable recording medium can be provided asso-called packaged software.

Incidentally, in addition to being installed from a removable recordingmedium as described above onto a computer, the program is transferred byradio from a download site to a computer or transferred by wire to acomputer via networks such as a LAN (Local Area Network), the Internet,and the like. The computer can receive the program transferred in such amanner, and install the program onto a recording medium such as abuilt-in hard disk or the like.

In addition, in the foregoing embodiment, description has been made of acase where the tap length and the filtering object pixel range asparameter values are set according to the block sizes of adjacent blocksadjacent to each other at a block boundary. However, when at least oneof the adjacent blocks has an extended block size larger than apredetermined block size, the filter setting section may set the valueof block boundary strength data used for filtering to an extended valueto be able to obtain an image of excellent image quality with a reducedblock distortion. For example, the filter setting section sets the valueof block boundary strength data used for filtering larger as the blocksize of the adjacent block is increased, to reduce block distortion.

The size of processing units of the deblocking filter or the size ofmacroblocks is not limited to the examples described in the presentspecification, but may be other sizes. For example, whereas the size ofmacroblocks of H.264/AVC is fixed at 16×16 pixels, the size of codingunits of HEVC can be dynamically specified for each sequence. A codingunit of HEVC is referred to also as a coding tree block. A coding unithaving a maximum size is referred to as a largest coding unit (LUC). Acoding unit having a minimum size is referred to as a smallest codingunit (SCU). A range of sizes of usable coding units is defined byspecifying the sizes of the LCU and the SCU in a sequence parameter setas a part of image compression information. Further, the sizes of codingunits used in individual sequences are identified by specifying thevalue of split flag.

When a coding unit has a square shape, the size of one side is expressedby a power of two. Coding units can be further divided into predictionunits (PU) as processing units of intra prediction and inter prediction.In addition, coding units can be divided into transform units (TU) asprocessing units of an orthogonal transform. HEVC allows the use oftransform units having sizes of 16×16 pixels and 32×32 pixels inaddition to 4×4 pixels and 8×8 pixels. The term “block” in the presentspecification includes the concept of macroblocks, coding units,prediction units, transform units, or various other units.

The sizes of blocks may be fixed or may change dynamically. Block sizesare not limited to the examples described in the present specification,but may be other sizes. A similar method can be used for blocks of 16×16pixels or smaller, such as block sizes of 4, 8, and 16, for example.Also in this case, it suffices to set the tap length of the filterlonger or set the filtering object pixel range wider as the block sizeis increased.

The present technology is applicable to not only cases of squares havingblock sizes of 4×4 pixels, 8×8 pixels, 16×16 pixels, and 32×32 pixelsbut also cases of non-squares, which are 8×2 pixels, 2×8 pixels, 16×4pixels, 4×16 pixels, 32×8 pixels, and 8×32 pixels. In this case, the taplength of filtering or the filtering object pixel range can be setaccording to the block sizes on adjacent sides of adjacent blocks. Inaddition, the tap length of filtering or the filtering object pixelrange can be set according to the block sizes on unadjacent sides of theadjacent blocks. Further, whether to apply the block sizes on theadjacent sides or whether to apply the block sizes on the unadjacentsides can be adaptively selected according to the shapes and block sizesof the blocks.

In addition, a method of transmitting the information used for theprocessing of the deblocking filter from the coding side to the decodingside is not limited to a method of multiplexing these pieces ofinformation in the header of the coded stream. For example, these piecesof information may be transmitted or recorded as separate dataassociated with the coded bit stream without being multiplexed in thecoded bit stream. The term “associated” in this case means that an imageincluded in the bit stream (which image may be a part of an image suchas a slice, a block, or the like) and information corresponding to theimage in question can be linked to each other at a time of decoding.That is, the information may be transmitted on a different transmissionline from that of the image (or the bit stream). In addition, theinformation may be recorded on a different recording medium (or in adifferent recording area in a same recording medium) from that of theimage (or the bit stream). Further, the information and the image (orthe bit stream) may be associated with each other in arbitrary unitssuch as a plurality of frames, one frame, or a part within a frame, forexample.

<7. Examples of Application>

The image coding device 10 and the image decoding device 50 according tothe foregoing embodiment can be applied to various electronic devicesincluding transmitters or receivers in satellite broadcasting, wirebroadcasting such as cable TV, distribution on the Internet,distribution to terminals by cellular communication, and the like,recording devices for recording images onto media such as optical disks,magnetic disks, and flash memories, or reproducing devices forreproducing the images from these storage media. Four examples ofapplication will be described in the following.

[7-1. First Example of Application]

FIG. 13 shows an example of a schematic configuration of a televisiondevice to which the foregoing embodiment is applied. A television device90 includes an antenna 901, a tuner 902, a demultiplexer 903, a decoder904, a video signal processing block 905, a display block 906, an audiosignal processing block 907, a speaker 908, an external interface 909, acontrol block 910, a user interface 911, and a bus 912.

The tuner 902 extracts a signal of a desired channel from a broadcastsignal received via the antenna 901, and demodulates the extractedsignal. The tuner 902 then outputs a coded bit stream obtained by thedemodulation to the demultiplexer 903. That is, the tuner 902 has afunction as transmitting means in the television device 90 for receivinga coded stream having a coded image.

The demultiplexer 903 separates a video stream and an audio stream of aprogram as a viewing object from the coded bit stream, and outputs eachof the separated streams to the decoder 904. In addition, thedemultiplexer 903 extracts auxiliary data of an EPG (Electronic ProgramGuide) or the like from the coded bit stream, and supplies the extracteddata to the control block 910. Incidentally, when the coded bit streamis scrambled, the demultiplexer 903 may perform descrambling.

The decoder 904 decodes the video stream and the audio stream input fromthe demultiplexer 903. The decoder 904 then outputs video data generatedby the decoding process to the video signal processing block 905. Inaddition, the decoder 904 outputs audio data generated by the decodingprocess to the audio signal processing block 907.

The video signal processing block 905 reproduces the video data inputfrom the decoder 904, and makes video displayed on the display block906. The video signal processing block 905 may also make an applicationscreen supplied via a network displayed on the display block 906. Thevideo signal processing block 905 may also subject the video data to anadditional process such as noise removal, for example, according tosettings. The video signal processing block 905 may further generate forexample a GUI (Graphical User Interface) image of a menu, a button, acursor, or the like, and superimpose the generated image on the outputimage.

The display block 906 is driven by a driving signal supplied from thevideo signal processing block 905. The display block 906 displays avideo or an image on a video screen of a display device (for example aliquid crystal display, a plasma display, or an OLED).

The audio signal processing block 907 subjects the audio data input fromthe decoder 904 to a reproducing process such as D/A conversion,amplification, and the like, and makes audio output from the speaker908. The audio signal processing block 907 may also subject the audiodata to an additional process such as noise removal. The externalinterface 909 is an interface for connecting the television device 90and an external device or a network. For example, a video stream or anaudio stream received via the external interface 909 may be decoded bythe decoder 904. That is, the external interface 909 also has a functionas transmitting means in the television device 90 for receiving a codedstream having a coded image.

The control block 910 has a processor such as a CPU (Central ProcessingUnit), and a memory such as a RAM (Random Access Memory) and a ROM (ReadOnly Memory). The memory stores a program executed by the CPU, programdata, EPG data, data obtained via a network, and the like. The programstored by the memory is, for example, read and executed by the CPU at atime of starting the television device 90. By executing the program, theCPU controls an operation of the television device 90 according to anoperating signal input from the user interface 911, for example.

The user interface 911 is connected to the control block 910. The userinterface 911 has, for example, a button and a switch for a user tooperate the television device 90, a portion for receiving a remotecontrol signal, and the like. The user interface 911 detects anoperation by the user via these constituent elements, generates anoperating signal, and outputs the generated operating signal to thecontrol block 910.

The bus 912 interconnects the tuner 902, the demultiplexer 903, thedecoder 904, the video signal processing block 905, the audio signalprocessing block 907, the external interface 909, and the control block910.

In the thus configured television device 90, the decoder 904 has thefunctions of the image decoding device 50 according to the foregoingembodiment. A range to which to apply the deblocking filter can bethereby determined more appropriately at a time of decoding an image inthe television device 90, so that image quality can be improved.

[7-2. Second Example of Application]

FIG. 14 shows an example of a schematic configuration of a portabletelephone to which the foregoing embodiment is applied. A portabletelephone 920 includes an antenna 921, a communicating block 922, anaudio codec 923, a speaker 924, a microphone 925, a camera block 926, animage processing block 927, a demultiplexing block 928, a recording andreproducing block 929, a display block 930, a control block 931, anoperating block 932, and a bus 933.

The antenna 921 is connected to the communicating block 922. The speaker924 and the microphone 925 are connected to the audio codec 923. Theoperating block 932 is connected to the control block 931. The bus 933interconnects the communicating block 922, the audio codec 923, thecamera block 926, the image processing block 927, the demultiplexingblock 928, the recording and reproducing block 929, the display block930, and the control block 931.

The portable telephone 920 performs operations such as transmitting andreceiving audio signals, transmitting and receiving electronic mails orimage data, taking an image, recording data, and the like in variousoperation modes including a voice call mode, a data communication mode,a photographing mode, and a videophone mode.

In the voice call mode, an analog audio signal generated by themicrophone 925 is supplied to the audio codec 923. The audio codec 923converts the analog audio signal into audio data, subjects the convertedaudio data to A/D conversion, and compresses the audio data. The audiocodec 923 then outputs the audio data after the compression to thecommunicating block 922. The communicating block 922 subjects the audiodata to coding and modulation to generate a transmission signal. Thecommunicating block 922 then transmits the generated transmission signalto a base station (not shown) via the antenna 921. In addition, thecommunicating block 922 subjects a radio signal received via the antenna921 to amplification and frequency conversion to obtain a receivedsignal. Then, the communicating block 922 generates audio data bydemodulating and decoding the received signal, and outputs the generatedaudio data to the audio codec 923. The audio codec 923 decompresses theaudio data and subjects the audio data to D/A conversion to generate ananalog audio signal. The audio codec 923 then supplies the generatedaudio signal to the speaker 924 to make audio output.

In addition, in the data communication mode, for example, the controlblock 931 generates text data constituting an electronic mail accordingto operation by the user via the operating block 932. The control block931 also makes the text displayed on the display block 930. In addition,the control block 931 generates electronic mail data according to atransmission instruction from the user via the operating block 932, andoutputs the generated electronic mail data to the communicating block922. The communicating block 922 subjects the electronic mail data tocoding and modulation to generate a transmission signal. Thecommunicating block 922 then transmits the generated transmission signalto the base station (not shown) via the antenna 921. The communicatingblock 922 also subjects a radio signal received via the antenna 921 toamplification and frequency conversion to obtain a received signal.Then, the communicating block 922 reconstructs electronic mail data bydemodulating and decoding the received signal, and outputs thereconstructed electronic mail data to the control block 931. The controlblock 931 makes the contents of the electronic mail displayed on thedisplay block 930, and makes the electronic mail data stored on astorage medium of the recording and reproducing block 929.

The recording and reproducing block 929 has an arbitrary readable andwritable storage medium. For example, the storage medium may be astorage medium of a built-in type such as a RAM, a flash memory, or thelike, or may be a storage medium of an external loading type such as ahard disk, a magnetic disk, a magneto-optical disk, an optical disk, aUSB memory, a memory card, or the like.

In addition, in the photographing mode, for example, the camera block926 generates image data by imaging a subject, and outputs the generatedimage data to the image processing block 927. The image processing block927 codes the image data input from the camera block 926, and makes acoded stream stored on the storage medium of the recording andreproducing block 929.

In addition, in the videophone mode, for example, the demultiplexingblock 928 multiplexes a video stream coded by the image processing block927 and an audio stream input from the audio codec 923, and outputs themultiplexed stream to the communicating block 922. The communicatingblock 922 subjects the stream to coding and modulation to generate atransmission signal. The communicating block 922 then transmits thegenerated transmission signal to the base station (not shown) via theantenna 921. The communicating block 922 also subjects a radio signalreceived via the antenna 921 to amplification and frequency conversionto obtain a received signal. The transmission signal and the receivedsignal can include a coded bit stream. Then, the communicating block 922reconstructs a stream by demodulating and decoding the received signal,and outputs the reconstructed stream to the demultiplexing block 928.The demultiplexing block 928 separates a video stream and an audiostream from the input stream, and outputs the video stream to the imageprocessing block 927 and the audio stream to the audio codec 923. Theimage processing block 927 decodes the video stream to generate videodata. The video data is supplied to the display block 930. The displayblock 930 displays a series of images. The audio codec 923 decompressesthe audio stream and subjects the audio stream to D/A conversion togenerate an analog audio signal. The audio codec 923 then supplies thegenerated audio signal to the speaker 924 to make audio output.

In the thus configured portable telephone 920, the image processingblock 927 has the functions of the image coding device 10 and the imagedecoding device 50 according to the foregoing embodiment. A range towhich to apply the deblocking filter can be thereby determined moreappropriately at times of coding and decoding an image in the portabletelephone 920, so that image quality can be improved.

[7-3. Third Example of Application]

FIG. 15 shows an example of a schematic configuration of a recording andreproducing device to which the foregoing embodiment is applied. Arecording and reproducing device 940 for example codes audio data andvideo data of a received broadcast program and records the audio dataand the video data on a recording medium. In addition, the recording andreproducing device 940 may for example code audio data and video dataobtained from another device and record the audio data and the videodata on the recording medium. In addition, the recording and reproducingdevice 940 for example reproduces data recorded on the recording mediumon a monitor and a speaker according to an instruction from a user. Atthis time, the recording and reproducing device 940 decodes audio dataand video data.

The recording and reproducing device 940 includes a tuner 941, anexternal interface 942, an encoder 943, an HDD (Hard Disk Drive) 944, adisk drive 945, a selector 946, a decoder 947, an OSD (On-ScreenDisplay) 948, a control block 949, and a user interface 950.

The tuner 941 extracts a signal of a desired channel from a broadcastsignal received via an antenna (not shown), and demodulates theextracted signal. The tuner 941 then outputs a coded bit stream obtainedby the demodulation to the selector 946. That is, the tuner 941 has afunction as transmitting means in the recording and reproducing device940.

The external interface 942 is an interface for connecting the recordingand reproducing device 940 with an external device or a network. Theexternal interface 942 may be for example an IEEE 1394 interface, anetwork interface, a USB interface, a flash memory interface, or thelike. For example, video data and audio data received via the externalinterface 942 is input to the encoder 943. That is, the externalinterface 942 has a function as transmitting means in the recording andreproducing device 940.

The encoder 943 codes the video data and the audio data input from theexternal interface 942 when the video data and the audio data are notcoded. The encoder 943 then outputs a coded bit stream to the selector946.

The HDD 944 records a coded bit stream having the content data of video,audio, and the like compressed therein, various kinds of programs, andother data on an internal hard disk. The HDD 944 also reads these piecesof data from the hard disk at a time of reproduction of the video andthe audio.

The disk drive 945 records and reads data on and from a recording mediumloaded therein. The recording medium loaded in the disk drive 945 may befor example a DVD disk (a DVD-Video, a DVD-RAM, a DVD-R, a DVD-RW, aDVD+R, a DVD+RW, or the like), a Blu-ray (registered trademark) disk, orthe like.

At a time of recording of video and audio, the selector 946 selects thecoded bit stream input from the tuner 941 or the encoder 943, andoutputs the selected coded bit stream to the HDD 944 or the disk drive945. In addition, at a time of reproduction of video and audio, theselector 946 outputs the coded bit stream input from the HDD 944 or thedisk drive 945 to the decoder 947.

The decoder 947 decodes the coded bit stream, and generates video dataand audio data. The decoder 947 then outputs the generated video data tothe OSD 948. In addition, the decoder 904 outputs the generated audiodata to an external speaker.

The OSD 948 reproduces the video data input from the decoder 947, anddisplays video. The OSD 948 may also superimpose for example a GUI imageof a menu, a button, a cursor, or the like on the displayed video.

The control block 949 has a processor such as a CPU and a memory such asa RAM and a ROM. The memory stores a program executed by the CPU,program data, and the like. The program stored by the memory is, forexample, read and executed by the CPU at a time of starting therecording and reproducing device 940. By executing the program, the CPUcontrols the operation of the recording and reproducing device 940according to an operating signal input from the user interface 950, forexample.

The user interface 950 is connected to the control block 949. The userinterface 950 has, for example, a button and a switch for a user tooperate the recording and reproducing device 940, a portion forreceiving a remote control signal, and the like. The user interface 950detects an operation by the user via these constituent elements,generates an operating signal, and outputs the generated operatingsignal to the control block 949.

In the thus configured recording and reproducing device 940, the encoder943 has the functions of the image coding device 10 according to theforegoing embodiment. In addition, the decoder 947 has the functions ofthe image decoding device 50 according to the foregoing embodiment. Arange to which to apply the deblocking filter can be thereby determinedmore appropriately at times of coding and decoding an image in therecording and reproducing device 940, so that image quality can beimproved.

[7-4. Fourth Example of Application]

FIG. 16 shows an example of a schematic configuration of an imagingdevice to which the foregoing embodiment is applied. An imaging device960 generates an image by imaging a subject, codes image data, andrecords the image data on a recording medium.

The imaging device 960 includes an optical block 961, an imaging block962, a signal processing block 963, an image processing block 964, adisplay block 965, an external interface 966, a memory 967, a mediadrive 968, an OSD 969, a control block 970, a user interface 971, and abus 972.

The optical block 961 is connected to the imaging block 962. The imagingblock 962 is connected to the signal processing block 963. The displayblock 965 is connected to the image processing block 964. The userinterface 971 is connected to the control block 970. The bus 972interconnects the image processing block 964, the external interface966, the memory 967, the media drive 968, the OSD 969, and the controlblock 970.

The optical block 961 has a focusing lens, a diaphragm mechanism, andthe like. The optical block 961 forms an optical image of a subject onan imaging surface of the imaging block 962. The imaging block 962 has aCCD or CMOS image sensor or the like. The imaging block 962 converts theoptical image formed on the imaging surface into an image signal as anelectric signal by photoelectric conversion. The imaging block 962 thenoutputs the image signal to the signal processing block 963.

The signal processing block 963 subjects the image signal input from theimaging block 962 to various camera signal processing such as kneecorrection, gamma correction, color correction, and the like. The signalprocessing block 963 outputs image data after the camera signalprocessing to the image processing block 964.

The image processing block 964 codes the image data input from thesignal processing block 963 to generate coded data. The image processingblock 964 then outputs the generated coded data to the externalinterface 966 or the media drive 968. The image processing block 964also decodes coded data input from the external interface 966 or themedia drive 968 to generate image data. The image processing block 964then outputs the generated image data to the display block 965. Theimage processing block 964 may also output the image data input from thesignal processing block 963 to the display block 965 to display theimage. The image processing block 964 may also superimpose data fordisplay which data is obtained from the OSD 969 on the image output tothe display block 965.

The OSD 969 generates for example a GUI image of a menu, a button, acursor, or the like, and outputs the generated image to the imageprocessing block 964.

The external interface 966 is configured as a USB input-output terminal,for example. The external interface 966 connects the imaging device 960to a printer at a time of printing of an image, for example. Inaddition, the external interface 966 is connected with a drive asrequired. A removable medium such as a magnetic disk or an optical disk,for example, is loaded into the drive. A program read from the removablemedium can be installed on the imaging device 960. Further, the externalinterface 966 may be configured as a network interface connected to anetwork such as a LAN, the Internet, or the like. That is, the externalinterface 966 has a function as transmitting means in the imaging device960.

A recording medium loaded into the media drive 968 may be an arbitraryreadable and writable removable medium such as a magnetic disk, amagneto-optical disk, an optical disk, or a semiconductor memory, forexample. In addition, a recording medium may be mounted in the mediadrive 968 in a fixed manner to form a nonportable storage portion suchas a built-in type hard disk drive or an SSD (Solid State Drive), forexample.

The control block 970 has a processor such as a CPU and a memory such asa RAM and a ROM. The memory stores a program executed by the CPU,program data, and the like. The program stored by the memory is, forexample, read and executed by the CPU at a time of starting the imagingdevice 960. By executing the program, the CPU controls the operation ofthe imaging device 960 according to an operating signal input from theuser interface 971, for example.

The user interface 971 is connected to the control block 970. The userinterface 971 has, for example, a button and a switch for a user tooperate the imaging device 960, and the like. The user interface 971detects an operation by the user via these constituent elements,generates an operating signal, and outputs the generated operatingsignal to the control block 970.

In the thus configured imaging device 960, the image processing block964 has the functions of the image coding device 10 and the imagedecoding device 50 according to the foregoing embodiment. A range towhich to apply the deblocking filter can be thereby determined moreappropriately at times of coding and decoding an image in the imagingdevice 960, so that image quality can be improved.

Further, the present technology should not be construed as being limitedto the foregoing embodiments. The embodiments disclose the presenttechnology in an illustrative form. It is obvious that modifications andsubstitutions in the embodiments can be made by those skilled in the artwithout departing from the spirit of the present technology. That is, inorder to determine the spirit of the present technology, claims are tobe considered.

Incidentally, the present technology can also adopt the followingconstitutions.

(1) An image processing device including:

a decoding section for decoding image data coded in each block;

a filter for applying filtering for removing block distortion to thedecoded image data decoded by the decoding section; and

a filter setting section for setting, according to block sizes ofadjacent blocks adjacent at a block boundary, a tap length of thefiltering for the block boundary or a filtering object pixel range as anobject of the filtering.

(2) The image processing device according to (1), wherein the filtersetting section sets the tap length to an extended length when at leastone of the adjacent blocks is extended to a size larger than apredetermined block size.

(3) The image processing device according to (2), wherein the filtersetting section sets the tap length of the filter longer as the blocksize of the adjacent block is increased.

(4) The image processing device according to any one of (1) to (3),wherein the filter setting section sets the filtering object pixel rangeto an extended width when at least one of the adjacent blocks isextended to a size larger than the predetermined block size.

(5) The image processing device according to any one of (1) to (4),wherein the filter setting section sets the filtering object pixel rangewider as the block size of the adjacent block is increased.

(6) The image processing device according to any one of (1) to (5),wherein the filter setting section sets a value of block boundarystrength data used for the filtering to an extended value when at leastone of the adjacent blocks is extended to a size larger than thepredetermined block size.

(7) The image processing device according to (6), wherein the filtersetting section sets the value of the block boundary strength data usedfor the filtering larger as the block size of the adjacent block isincreased.

(8) The image processing device according to any one of (1) to (7),wherein the filter setting section sets the tap length of the filteringor the filtering object pixel range according to the block sizes onadjacent sides of the adjacent blocks.

(9) The image processing device according to any one of (1) to (8),wherein the filter setting section sets the tap length of the filteringand the filtering object pixel range according to case classificationcorresponding to the block sizes of the adjacent blocks.

(10) The image processing device according to (9), wherein the caseclassification is a case where the adjacent blocks both have apredetermined block size or smaller and a case where at least one of theadjacent blocks is extended to a size larger than the predeterminedblock size.

(11) The image processing device according to (10), wherein the filtersetting section performs the case classification into a case where theadjacent blocks are 16×16 pixels or smaller, a case where at least oneof the two blocks is larger than 16×16 pixels and both are 32×32 pixelsor smaller, and a case where at least one of the two blocks is largerthan 32×32 pixels.

(12) The image processing device according to any one of (1) to (11),wherein the block sizes are prediction block sizes as processing unitswhen intra prediction or inter prediction is performed.

(13) The image processing device according to any one of (1) to (12),wherein the block sizes are transform sizes as processing units when anorthogonal transform is performed.

(14) The image processing device according to any one of (2) to (13),wherein the predetermined block size is a macroblock size of anH.264/AVC standard.

(15) The image processing device according to any one of (1) to (14),wherein the filter setting section sets the tap length or the filteringobject pixel range according to whether the decoded image data is imagedata for generating a predictive image or image data for image display.

INDUSTRIAL APPLICABILITY

The image processing device and the image processing method according tothe present technology can provide an image of excellent image qualitywith a reduced block distortion. The present technology is thereforesuitable for image coding devices, image decoding devices, and the likeused when image information (bit stream) obtained by performing codingin block units as in MPEG, H.26×, and the like is transmitted andreceived via network media such as satellite broadcasting, cable TV, theInternet, portable telephones, and the like or when the imageinformation is processed on storage media such as optical and magneticdisks, flash memories, and the like.

EXPLANATION OF REFERENCE NUMERALS

-   -   10 . . . Image coding device, 11 . . . A/D converting section,        12, 57 . . . Picture rearrangement buffer, 13 . . . Subtracting        section, 14 . . . Orthogonal transform section, 15 . . .        Quantizing section, 16 . . . Lossless coding section, 17, 51 . .        . Storage buffer, 18 . . . Rate controlling section, 21, 53 . .        . Dequantizing section, 22, 54 . . . Inverse orthogonal        transform section, 23, 55 . . . Adding section, 24, 56 . . .        Deblocking filter, 25, 61 . . . Frame memory, 26, 62, 65 . . .        Selector, 31, 63 . . . Intra prediction section, 32 . . . Motion        prediction and compensation section, 33 . . . Predictive image        and optimum mode selecting section, 41, 71 . . . Filter setting        section, 50 . . . Image decoding device, 52 . . . Lossless        decoding section, 58 . . . D/A converting section, 64 . . .        Motion compensation section, 90 . . . Television device, 92 . .        . Portable telephone, 94 . . . Recording and reproducing device,        96 . . . Imaging device, 241 . . . Filter strength determining        portion, 242 . . . Filtering portion, 411 . . . Block size        buffer, 412 . . . Parameter value generating portion, 901, 921 .        . . Antenna, 902, 941 . . . Tuner, 903 . . . Demultiplexer, 904,        947 . . . Decoder, 905 . . . Video signal processing block, 906        . . . Display block, 907 . . . Audio signal processing block,        908 . . . Speaker, 909, 942, 966 . . . External interface block,        910, 931, 949, 970 . . . Control block, 911, 932, 971 . . . User        interface block, 912, 933, 972 . . . Bus, 922 . . .        Communicating block, 923 . . . Audio codec, 924 . . . Speaker,        925 . . . Microphone, 926 . . . Camera block, 927 . . . Image        processing block, 928 . . . Demultiplexing block, 929 . . .        Recording and reproducing block, 930 . . . Display block, 943 .        . . Encoder, 944 . . . HDD block, 945 . . . Disk drive, 948, 969        . . . OSD block, 961 . . . Optical block, 962 . . . Imaging        block, 963 . . . Camera signal processing block, 964 . . . Image        data processing block, 965 . . . Display block, 967 . . . Memory        block, 968 . . . Media drive

1-18. (canceled)
 19. An image processing device comprising: a decodingsection for decoding image data coded in each of blocks; a filter forapplying filtering for removing block distortion to the decoded imagedata decoded by the decoding section; and a filter setting section forsetting, according to block sizes of adjacent blocks adjacent to eachother at a block boundary and shapes of the adjacent blocks, a taplength of the filtering for the block boundary or a filtering objectpixel range as an object of the filtering.
 20. The image processingdevice according to claim 19, wherein, when the block sizes on adjacentsides of the adjacent blocks are extended block sizes larger than apredetermined block size, the filter setting section sets the tap lengthlonger than a tap length set at the predetermined block size or sets thefiltering object pixel range wider than a filtering object pixel rangeset at the predetermined block size.
 21. The image processing deviceaccording to claim 19, wherein, when a block of the adjacent blocks isextended to a size larger than the predetermined block size, the filtersetting section sets a value of block boundary strength data used forthe filtering larger than a value set at the predetermined block size.22. The image processing device according to claim 19, wherein the blocksizes are prediction block sizes as processing units when intraprediction or inter prediction is performed.
 23. The image processingdevice according to claim 19, wherein the block sizes are transformsizes as processing units when an orthogonal transform is performed. 24.The image processing device according to claim 19, wherein thepredetermined block size is a macroblock size of an H.264/AVC standard.25. The image processing device according to claim 19, wherein thefilter setting section sets the tap length or the filtering object pixelrange according to whether the decoded image data is image data forgenerating a predictive image or image data for image display.
 26. Theimage processing device according to claim 19, wherein the filtersetting section sets the tap length of the filtering or the filteringobject pixel range according to the block sizes on adjacent sides of theadjacent blocks.
 27. The image processing device according to claim 19,wherein the filter setting section sets the tap length of the filteringor the filtering object pixel range according to the block sizes onunadjacent sides of the adjacent blocks.
 28. The image processing deviceaccording to claim 19, wherein the adjacent blocks are non-squareblocks.
 29. An image processing method comprising: decoding image datacoded in each of blocks; applying filtering for removing blockdistortion to the decoded image data decoded in the decoding; andsetting, according to block sizes of adjacent blocks adjacent at a blockboundary and shapes of the adjacent blocks, a tap length of thefiltering for the block boundary or a filtering object pixel range as anobject of the filtering.
 30. An image processing device comprising: afilter for applying filtering for removing block distortion to decodedimage data obtained by locally decoding image data resulting from anorthogonal transform and quantization; a filter setting section forsetting, according to block sizes of adjacent blocks adjacent at a blockboundary and shapes of the adjacent blocks, a tap length of thefiltering for the block boundary or a filtering object pixel range as anobject of the filtering; and a coding section for performing coding ineach image data block using the decoded image data resulting from thefiltering by the filter.
 31. An image processing method comprising:applying filtering for removing block distortion to decoded image dataobtained by locally decoding image data resulting from an orthogonaltransform and quantization; setting, according to block sizes ofadjacent blocks adjacent at a block boundary and shapes of the adjacentblocks, a tap length of the filtering for the block boundary or afiltering object pixel range as an object of the filtering; andperforming coding in each image data block using the decoded image dataresulting from the filtering in the filter.